Vaccines: Many Beliefs Study 3: Overparenting deep dive

This is a document where we do a “deep dive” on the “Overparenting” scale, which is one of the scales we are least able to predict movement in after the Disease Risk intervention - and a case that we think might be involved meaningful interactions with other scales (e.g., Disease Severity, Medical Skepticism).

Reliability

Let’s look at the observed reliability of the Overparenting scale at pre-test:

Reliability analysis   
Call: alpha(x = .)

  raw_alpha std.alpha G6(smc) average_r S/N  ase   mean sd median_r
      0.71      0.71    0.71      0.33 2.5 0.02 -0.016  1     0.31

 lower alpha upper     95% confidence boundaries
0.67 0.71 0.75 

 Reliability if an item is dropped:
           raw_alpha std.alpha G6(smc) average_r S/N alpha se var.r med.r
overpar_1       0.66      0.66    0.63      0.33 2.0    0.025 0.014  0.31
overpar_2       0.64      0.63    0.60      0.30 1.7    0.026 0.022  0.29
overpar_3       0.61      0.62    0.59      0.29 1.6    0.029 0.023  0.24
overpar_4r      0.73      0.73    0.70      0.41 2.7    0.019 0.019  0.46
overpar_5r      0.67      0.68    0.65      0.35 2.1    0.024 0.035  0.36

 Item statistics 
             n raw.r std.r r.cor r.drop   mean  sd
overpar_1  493  0.68  0.69  0.60   0.48  0.728 1.4
overpar_2  493  0.73  0.74  0.67   0.56  0.998 1.3
overpar_3  493  0.79  0.77  0.70   0.59 -0.041 1.8
overpar_4r 493  0.56  0.56  0.37   0.31 -0.434 1.5
overpar_5r 493  0.66  0.66  0.53   0.45 -1.333 1.5

Non missing response frequency for each item
             -3   -2   -1    0    1    2    3 miss
overpar_1  0.01 0.05 0.12 0.21 0.32 0.18 0.10    0
overpar_2  0.01 0.04 0.07 0.21 0.30 0.23 0.14    0
overpar_3  0.07 0.17 0.24 0.13 0.15 0.15 0.10    0
overpar_4r 0.05 0.19 0.29 0.22 0.12 0.08 0.04    0
overpar_5r 0.23 0.29 0.25 0.12 0.05 0.04 0.02    0

Not spectacular, but there don’t appear to be items that would dramatically improve it.

Let’s look at the observed reliability of the Overparenting scale at post-test:

Reliability analysis   
Call: alpha(x = .)

  raw_alpha std.alpha G6(smc) average_r S/N   ase mean  sd median_r
      0.76      0.76    0.75      0.39 3.2 0.017 0.11 1.1     0.37

 lower alpha upper     95% confidence boundaries
0.73 0.76 0.8 

 Reliability if an item is dropped:
           raw_alpha std.alpha G6(smc) average_r S/N alpha se  var.r med.r
overpar_1       0.72      0.72    0.69      0.39 2.5    0.021 0.0143  0.37
overpar_2       0.71      0.71    0.67      0.38 2.5    0.021 0.0089  0.37
overpar_3       0.69      0.69    0.66      0.36 2.3    0.023 0.0156  0.30
overpar_4r      0.75      0.75    0.71      0.43 3.0    0.018 0.0129  0.45
overpar_5r      0.73      0.73    0.69      0.40 2.7    0.020 0.0178  0.39

 Item statistics 
             n raw.r std.r r.cor r.drop  mean  sd
overpar_1  493  0.71  0.73  0.63   0.54  0.75 1.4
overpar_2  493  0.73  0.73  0.66   0.56  1.01 1.5
overpar_3  493  0.79  0.77  0.69   0.60  0.23 1.8
overpar_4r 493  0.65  0.65  0.52   0.45 -0.31 1.5
overpar_5r 493  0.71  0.70  0.60   0.51 -1.12 1.5

Non missing response frequency for each item
             -3   -2   -1    0    1    2    3 miss
overpar_1  0.02 0.03 0.15 0.17 0.33 0.21 0.09    0
overpar_2  0.01 0.05 0.10 0.15 0.27 0.26 0.16    0
overpar_3  0.06 0.11 0.23 0.12 0.18 0.18 0.11    0
overpar_4r 0.05 0.15 0.31 0.20 0.13 0.11 0.04    0
overpar_5r 0.19 0.26 0.28 0.11 0.07 0.05 0.03    0

Again, not spectacular, though perhaps slightly better than at pre-test; no obvious items that should be dropped.

It’s possible that the lower-than-average reliability for this scale is one of the reasons it’s harder to predict - it’s the lowest reliability of all 14 scales at both pre- and post-test (though not dramatically lower than the Disease Rarity scale at pre-test, \(\alpha\) = 0.75, or the Naturalism scale at post-test, \(\alpha\) = 0.76).

Distributions of scores

Let’s look at the overall distribution of scores on the Overparenting scale, by condition and phase:

And at the difference scores:

Looks like mean scores were around the midpoint (0) at pre-test in both conditions, and increased at post-test among participants in the Disease Risk condition (but not the No Intervention condition). This increase in the Disease Risk condition seems to be on the order of 0.25 pts on our 7-point scale.

Correlations with other scales

Let’s look at the observed correlations between the Overparenting scale and other scales:

Joining, by = c("scaleA", "scaleB")
Joining, by = c("scaleA", "scaleB")
Joining, by = c("scaleA", "scaleB")

Joining, by = c("scaleA", "scaleB")
Joining, by = c("scaleA", "scaleB")
Joining, by = c("scaleA", "scaleB")

At pre-test, none of the other scales was strongy correlated with Overparenting among participants in either condition; the strongest were the two infant immune system beliefs scales (IIS: Limited Capacity and IIS: Weak).

In the No Intevention condition, these correlations seem to have increased at post-test relative to pre-test; this didn’t happen in the Disease Risk condition. The correlation between Overparenting and Medical Skepticism also seems to have increased in this condition from pre- to post-test in the No Intervention condition.

In the Disease Risk condition, a handful of correlations seem to have decreased from pre- to post-test, most notably all the correlations with vaccine beliefs, as well as the correlation with Medical Skepticism. Meanwhile, a different set of correlations seem to have increased from pre- to post-test, including correlations with Disease Severity (the target of the intervention), Vaccine Intention (the outcome of interest), and Vaccine Efficacy.

So it seems like just filling out these scales twice might have made participants align their beliefs about infant immune systems with their attitudes toward protective parenting (at least in the No Intervention condition) - but in addition to this, the intervention might have changed the relationship between participants’ beliefs about vaccines and their attitudes toward protective partenting. In particular, at pre-test, participants who more strongly endorsed what we’re calling “overparenting” also tended to think vaccines were more dangerous, less effective, more of a strain on the infant immune system, and more likely to contain toxins, and were generally lower on our Vaccine Intent scale. But at post-test, these relationships no longer held - all of these correlations were pretty close to 0.

(Note that this way of thinking about things is at odds with our use of the same cognitive model connecting all of these beliefs - I need to think harder about this.)

Quantile splits

Let’s split people into groups by their scores on the Overparenting, Medical Skepticism, and Disease Severity scores, since these were possible interactions that we identified in person. (Note that we could also use our investigations of the changing correlations among scales to do further investigations like this…)

Overparenting and Disease Severity

What I had written down in our notes from our meeting on 2018-05-15 was that the Disease Risk intervention might be most effective for participants who were high in Overparenting and low or moderate on Disease Severity. This does seem to be the case - these are the top and middle rows for the right-most column.

Interestingly, this plot reveals that the intervention generally didn’t work for participants who started out as low in Overparenting (left-most column) - this is good to know… not sure yet what to make of it!

As we’d predict, the intervention didn’t do much for participants who started out as high in Disease Severity (bottom row).

Overparenting and Medical Skepticism

I also had written down in our notes that the Disease Risk intervention might be most effective for participants who were high in Overparenting and high in Medicial Skepticism. Again, this does seem to be the case - this would be the bottom right panel.

Note, however, that this might just be a main effect of Medical Skepticism: Participants who started out high in Medical Skepticism (bottom row) seem to have been particularly sensitive to the intervention, perhaps regardless of their Overparenting scores.

Interestingly, this plot reveals that the intervention generally didn’t work for participants who started out as low in Medical Skepticism (top row) - again, this is good to know, though again not sure yet what to make of it!

Modeling interactions via regression

Let’s model these potential interactions via regression. (Again, we could also do this for any of the additional relationships of interest revealed by the changes in correlations, above).

I started off with my go-to model of the intervention, score ~ phase * condition + (1|subject) (with phase and condition effect-coded), because it’s the one that always makes sense to me, but we should consider doing this with other models as well.

On the idea that we were interested in possible interactions involving Overparenting, Medicial Skepticism, and Disease Severity in particular, I created a new dataframe with pre-test scores for Overparenting, Medicial Skepticism, and Disease Severity (all mean-centered and standardized), and both pre- and post-test scores for Vaccine Intentions (also mean-centered and standardized). I think proceeded through a series of models of increasing complexity (though I know this isn’t a very sophisticated approach) - see the code for everything I did.

Somewhat surprisingly, the “best-fitting” model according to model comparison is the one with all of the interactions (I’ve marked parameters with \(t\) > 2 with an asterisk in the table below):

Parameter	Estimate	Std..Error	t.value
(Intercept)	0.02	0.03	0.71
phasepost	0.02	0.01	1.54
conditionDR	0.03	0.03	0.89
overpar	-0.17	0.03	-4.77	*
medSkept	-0.48	0.03	-14.91	*
diseaseSevere	0.39	0.03	12.14	*
phasepost:conditionDR	0.03	0.01	2.61	*
phasepost:overpar	0.02	0.01	1.39
conditionDR:overpar	-0.02	0.03	-0.47
phasepost:medSkept	0.02	0.01	1.99
conditionDR:medSkept	0.01	0.03	0.47
overpar:medSkept	-0.01	0.03	-0.21
phasepost:diseaseSevere	0.00	0.01	0.36
conditionDR:diseaseSevere	-0.05	0.03	-1.61
overpar:diseaseSevere	0.04	0.04	1.06
medSkept:diseaseSevere	0.09	0.03	3.23	*
phasepost:conditionDR:overpar	0.01	0.01	0.62
phasepost:conditionDR:medSkept	0.02	0.01	1.70
phasepost:overpar:medSkept	0.00	0.01	-0.09
conditionDR:overpar:medSkept	0.04	0.03	1.20
phasepost:conditionDR:diseaseSevere	0.01	0.01	0.59
phasepost:overpar:diseaseSevere	-0.01	0.01	-0.67
conditionDR:overpar:diseaseSevere	-0.04	0.04	-0.98
phasepost:medSkept:diseaseSevere	0.01	0.01	0.45
conditionDR:medSkept:diseaseSevere	-0.07	0.03	-2.35	*
overpar:medSkept:diseaseSevere	-0.06	0.03	-2.00	*
phasepost:conditionDR:overpar:medSkept	0.03	0.01	2.11	*
phasepost:conditionDR:overpar:diseaseSevere	0.00	0.01	-0.09
phasepost:conditionDR:medSkept:diseaseSevere	0.00	0.01	0.04
phasepost:overpar:medSkept:diseaseSevere	0.01	0.01	1.12
conditionDR:overpar:medSkept:diseaseSevere	-0.07	0.03	-2.50	*
phasepost:conditionDR:overpar:medSkept:diseaseSevere	-0.02	0.01	-1.46

I think the most interesting “significant” parameter here is:

• phasepost:conditionDR:overpar:medSkept: A 4-way interaction betwen phase, condition, Overparenting at pre-test, and Medicial Skepticism at pre-test. Without thinking too deeply about this yet, I think this means that the intervention effect (the interaction between phase and condition) depended on pre-test scores for Overparenting and Medicial Skepticism. This is consistent with the observations we made looking at the second set of quantile splits, above.

Also interesting to think about (later):

• conditionDR:medSkept:diseaseSevere: A 3-way interaction between condition, Medical Skepticism at pre-test, and Disease Severity at pre-test.
• conditionDR:overpar:medSkept:diseaseSevere: A 4-way interaction between condition, Overparenting at pre-test, Medicial Skepticism at pre-test, and Disease Severity at pre-test.

Take-aways

This all suggests to me that it would be worth it to try including a node for the interaction between Overparenting and Medical Skepticism (and perhaps a node for the interaction between Overparenting and Disease Severity) in our cognitive model. We’ve come up with many (post-hoc) reasons why this might be important, and our data suggest that it was.